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ABSTRACT 

Signal recovery from a given set of linear measurements us- 
ing a sparsity prior has been a major subject of research in 
recent years. In this model, the signal is assumed to have a 
sparse representation under a given dictionary. Most of the 
work dealing with this subject has focused on the reconstruc- 
tion of the signal's representation as the means for recovering 
the signal itself. This approach forced the dictionary to be 
of low coherence and with no linear dependencies between 
its columns. Recently, a series of contributions that focus on 
signal recovery using the analysis model find that linear de- 
pendencies in the analysis dictionary are in fact permitted and 
beneficial. In this paper we show theoretically that the same 
holds also for signal recovery in the synthesis case for the Iq- 
synthesis minimization problem. In addition, we demonstrate 
empirically the relevance of our conclusions for recovering 
the signal using an ^i-relaxation. 

Index Terms — Sparse representations, compressed sens- 
ing, analysis versus synthesis, inverse problems. 

1. INTRODUCTION 

Recovering a signal from a noisy set of linear measurements 
appears in many problems such as compressed sensing, im- 
age deblurring and super-resolution. In the basic setup an un- 
known signal xo G passes through a given linear transfor- 
mation M e M'"^''; an additive noise e G M™ contaminates 
the outcome, providing a set of linear measurements 



Mxo 



(1) 



The noise e can be assumed to be either an adversarial un- 
known bounded noise such that ||e|j2 < e or a random noise 
with a given distribution, e.g., zero-mean white Gaussian 
noise. In this paper we focus on the first case. The measure- 
ment matrix M is usually either rectangular with m < d or an 
ill-posed matrix with m = d. Thus, the information we have 
in the measurement vector y is not enough for recovering x, 
and additional prior knowledge is needed. 



The priors considered in this work are sparsity based ones 
[T] . These include two main paradigms for modeUng the sig- 
nal - the synthesis and the analysis schemes 0. In the syn- 
thesis framework the signal x is assumed to have a sparse 
representation a G M" under a given dictionary D G M*^^", 



Xo = Dao- 



(2) 
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Few words on our terminology: A vector a is said to be k- 
sparse if ||Q:||g < k, where ||-||q is the £o semi-norm that 
counts the number of its non-zero elements, and a signal x 
is said to have a fc-sparse representation if there exists a k- 
sparse a such that x = Da. The columns of D are referred 
to as atoms, the non-zero locations of a sparse vector a as the 
support T, the size of this support is |T|, and the sub-matrix 
of D with the atoms corresponding to T as Dt- Given that xq 
has a fc-sparse representation, we can recover it from its mea- 
surement y by looking for the sparsest vector a that satisfies 
||y — MDq:||2 < e (In the noiseless case, e = 0). 

The analysis framework models the signal differently. 
Given a matrix CI E Rp^'^ - the analysis dictionary - we 
are interested in the multiplication Ox, referred to as the 
analysis representation. A signal x is said to be £-cosparse 
if ||Ox||p < p-£, i.e., Jlx has £ zero elements. In this case 
we recover the signal by looking for the most cosparse vector 
X, i.e., the one with the largest number of zeros in Hx that 
satisfies j|y — Mx|j2 < e. More details can be found in mil. 

Unlike the analysis model which works in the signal do- 
main and thus recovers it directly, the synthesis one works in 
the representation domain and thus recovers the signal indi- 
rectly by recovering first its representation. Because of this, 
most of the work that studies theoretical guarantees for the 
synthesis framework give bounds for the representation's re- 
construction, or deal only with the case that D is unitary, for 
which the two problems coincide l5l l6l 171 l8l l9l [TOl [m [Til . 

Results for more general dictionaries also exist llT3l [141 . 
However, they require the dictionary to be highly incoherent 
and with no linear dependencies between small number of its 
columns. This requirement limits the types of dictionaries 
that can be used to model the signal. While this constraint 
is necessary for recovering the signal's representation, it is 
not clear that it is still required when our target is the signal. 
The goal of this paper is to answer this very question. We 
aim to show that highly-coherent and even linearly dependent 



atoms in D still enable a reliable recovery of the signal from 
its measurements. 

Recently, a series of works IfTTl |3] |4] [16] [17] showed that 
in the analysis framework, where we aim at recovering the 
signal directly, linear dependencies can be allowed in the 
(analysis) dictionary. These correlations were even favored 
as they were shown to improve the recovery performance of 
the different techniques analyzed in these works. 

Since linear dependencies are shown to give profit in the 
analysis framework, it is conjectured that the requirement for 
an incoherent dictionary for the signal recovery in the syn- 
thesis framework is unnecessary as well. A clue for this very 
property is given in lITSl . where the reconstruction conditions 
are presented in terms of D-RIP which is a property of the 
measurement matrix M for the synthesis model. However, 
the results in |[T5l are derived for signals from the analysis 
model, thus leaving our question unresolved as of yet. 

In this work we focus on the synthesis model and ask: Is it 
possible to have a recovery guarantee for the synthesis model 
for a dictionary exhibiting linear dependencies within small 
groups of its columns? In order to answer this question, we 
study the performance of the ideal -minimization problem 

a — argmin jjallQ s.t. ||y — MDa||2 < e. (3) 

a 

This is the core approach for recovering a signal that is known 
to have the synthesis sparsity prior The reconstructed signal 
in this technique is simply x = Da. In the noiseless case, ([3]l 
turns to be simply 

a = argmin llajlp s.t. y = MDcc. (4) 

a 

In this work we study the recovery performance of (|3} 
and We first provide uniqueness conditions for the sig- 
nal recovery in the noiseless case. Then we present stable 
reconstruction conditions for the noisy case where the noise 
is adversarial. This result is a particular case of the result pre- 
sented in |[T8l[T9l in which a more general form of the D-RIP 
property was proposed giving stable recovery guarantee for 
signals that come from a general union of subspaces model. 

The uniqueness and stability conditions that we present 
are generalization of previous results ll20] |5] H] |7] [8] |2T] that 
assume D to be incoherent. The contribution of this work is 
in providing reconstruction guarantees in the signal domain 
that do not pose incoherence requirement on the dictionary. 

The £o-minimization problem is known to be NP-hard 
II22II and several approximation techniques have been sug- 
gested toit|(20l|5]|9l[l0l[IT][l2l|23|2l. A future work 
should extend the existing representation recovery guarantees 
to the signal case. First steps in this direction have been taken 
for the CoSaMP algorithm in l25]. However, the theoretical 
study of this algorithm is impaired by the assumed existence 
of a near-optimal projection scheme in the signal domain, an 
assumption similar to the one posed in I?]. The existence of 
such a projection in the general case is still an open question. 



In addition to the theoretical study of the £o-problem, in 
this paper we show empirically that for the £1 -synthesis min- 
imization problem, a relaxed version of ((3) that replaces the 
^Q-semi-norm with an £i-norm, signal recovery is at hand in 
the cases where D is highly coherent and the representation 
recovery is impossible. Similar experiments has been per- 
formed in liZSl for the CoSaMP algorithm. 

The organization of this paper is as follows. In Section [2] 
we survey existing uniqueness and stability guarantees for 
the representation reconstruction using the £o-minimization 
problem. In Section [3] the previously presented results are 
extended for the signal reconstruction case, allowing general 
coherent dictionaries. Empirical reconstruction results for the 
^1 -minimization problem are presented in Section |4] compar- 
ing the representation and signal recovery in the case where 
D is highly coherent. In Section |5]the work is summarized. 

2. BACKGROUND - GUARANTEES FOR THE 
REPRESENTATION RECOVERY 

Two important properties that were explored for the Iq- 
problem are the uniqueness of its solution and the stabihty of 
the solution under bounded adversarial noise. In this section 
we survey the existing results for representation recovery. 
We present the results with no proofs, as they are already 
contained in those of Section[3]for signal recovery. 

2.1. Uniqueness 

Given a signal's representation ao with a cardinality k 
(!|cKo|lo ^ fc) we are interested to know whether it is the 
unique solution of In other words, whether there exists 
another representation oti 7^ otQ with a cardinality at most 
k such that MDcio — MDai. An answer for this question 
was given in 15] using the definition of the Spark of a matrix: 

Definition 2.1 (Definition 1 in |(5l) Given a matrix A we de- 
fine Spark(A) as the smallest possible number of columns 
from A that are linearly dependent. 

This definition provides us with a sharp uniqueness condition 
for the representation reconstruction: 

Tlieorem 2.2 (Corollary 3 in 0) Let y = MDccq. // 

jjaollg < Spark(]V[D)/2 then olq is the unique solution 
of®. 

Though this condition is sharp for finding the representation, 
it is not sharp at all, in terms of finding the signal itself. This 
can be demonstrated using the following simple example. Let 
us assume that D = [z, z, . . . , z], a dictionary with columns 
that are a duplicate of the same atom z. Clearly, the signal 
x=z can be represented by any of the atoms of D, which 
means that it has n different sparse representations, each with 
cardinality 1. Thus, for any measurement matrix M there is 



no unique solution to (|4|l. Indeed, we have Spark(MD) = 2 
and the uniqueness condition collapses to the trivial require- 
ment IIccoIIq = 0. However, it is clear that if our goal is to 
recover the signal x (i.e. Da) and not its representation a, 
then we can certainly have a uniqueness, as all the possible 
solutions to ^ lead to the same signal. Thus, we conclude 
that for the task of estimating the signal itself, the existing 
condition is not sharp and a better one should be explored. 

2.2. Stability 

In the case where noise exists in the measurement, the unique- 
ness of the solution is no longer the question since we cannot 
recover the original signal exactly. Instead, we ask whether 
the reconstruction is stable, i.e., whether the ^2-distance be- 
tween the estimated signal's representation and the original 
one is proportional to the noise power e. In order to establish 
that, we use the restricted isometry property (RIP) ||7] |8] . The 
RIP can be seen as an extension of the Spark, which allows 
noisy case analysis. 

Definition 2.3 A matrix A satisfies the RIP condition with 
parameter Sk if it is the smallest value that satisfies 

(1 - 5k) \\oc\\l < \\Aa\\l < (1 + 4) l|a|l2 (5) 
fijr any k-sparse vector a. 

The connection between the RIP and the Spark is the follow- 
ing. Given a matrix A, fc < Spark(A) if (5^ < 1 ||8l. Having 
the RIP it is straight forward to have a stability condition for 
the representation reconstruction using (|3). 

Tlieorem2.4 Let y = MDccq + e where \\e\\^ < e, MD 

satisfies the RIP condition with 62k and ||cto||o l£ k. If 62k < 
1 then the solution a of ^ is stable. More specifically, 

||ao-a|l2 < -=i^. (6) 

The same problem we had with the dictionary D = [z, z, . . . , z 
in the uniqueness case repeats also here because S2k < 1 im- 
plies 2k < Spark(MD). 

3. GUARANTEES FOR SIGNAL RECOVERY 

As mentioned before, though the Spark and RIP conditions 
for MD are sharp for the representation recovery, they are 
not designed for the signal recovery. In this section we use an 
extended Spark and RIP definitions that will serve better the 
signal recovery problem. The D-RIP ifTSll is used for having 
stable recovery conditions for the signal reconstruction. In a 
similar way, we propose a D-Spark property for the measure- 
ment matrix M, introducing a new uniqueness condition for 
the signal recovery. Note that the results in this section are 
generalization of the ones presented in the previous section 



for the signal's representation. As a general guideline, by set- 
ting the measurement matrix to be MD and the dictionary to 
be the identity, the results of this section coincide with those 
of the previous one. 

3.1. Uniqueness for Signal Recovery 

As in the representation case, we are interested to know when 
we can guarantee that a signal xq with a fc-sparse represen- 
tation under a matrix D is the unique solution of (|4|i. In 
other words, whether there exist another signal Xi 7^ xq with 
at most fc-sparse representation under D such that Mxo = 
Mxi. For this task we introduce the D-Spark, an extension 
of the Spark definition. 

Definition 3.1 Given a matrix M we define D-Spark(M) as 
the smallest possible number of columns from D, marked as 
T, such that range(DT) n Null(M) ^ {0}. 

In other words, for every set T with size |r| < D-Spark(M) 
we have range(D'r) n Null(M) = {0}, implying that for 
any vector v e M'^I, MD^v = if and only if D^v = 0. 
Note that this definition coincides with the one of the Spark 
for D=I. This can be observed by noticing that in this case 
Dt=It and thus MD^ simply chooses T columns from M. 
Thus, the above translates to the requirement that there is no 
subset of |T| columns in M that are linearly dependent. 

Having the D-Spark definition, we propose a uniqueness 
condition for the signal recovery. 

Theorem 3.2 Let y = Mxg where xo has a k-sparse rep- 
resentation otQ under T). Ifk < D-Spark(M)/2 then xo = 
Dot for a the minimizer of implying a perfect recovery. 

Proof: Let us assume the contrary, i.e., there exists a mini- 
mizer, a.1 for dUi such that Dai 7^ Dckq. Let us denote the 
support sets of cxq and a.i by Tq and Ti respectively. Since 
do is a feasible solution to ^ and ai is a minimizer, we 
have that \Ti\ < \Tq\ < fc. Thus, by definition Dai - 
Dao S rangc(L'TiuTo) where iTiUTol < 2fc. Noticing 
that MDai = MDao- by the constraint of dUi, we have 
Dai — Dao G Null(M). This contradicts the assumption 
fc < D-Spark(M)/2 because we get that Dai - Dao G 
range(DTiUTo) n Null(M) 7^ {0}, which means that D- 
Spark(M) < |Ti U TqI < 2fc. □ 

Unlike the uniqueness condition of the regular Spark, the 
one of the D-Spark allows linear dependencies within the dic- 
tionary D. Looking at the example from the previous section, 
for the dictionary D = [z, z, . . . , z], range(DT) = {/3z, /3 E 
R} for any non-empty support T. Thus, the uniqueness con- 
dition turns out to be z ^ Null(M). This means that the fam- 
ily of matrices M that guarantee uniqueness are the ones that 
have at least one row non-orthogonal to z. This is far stronger 
compared to earlier condition as discussed in Section |2] 



3.2. Stability for Signal Recovery 

Moving to the noisy case, we seek for a generalization of the 
RIP that provides us with guarantees for the signal recovery. 
For this task we use the D-RIP, as introduced in 1131 . 

Definition 3.3 A matrix M satisfies the D-RIP condition 
with parameter 5^ if it is the smallest value that satisfies 





(l-<5,^)||x||^<||Mx||^<(l + <5P)||x|| 



(7) 



for any vector x that has a k-sparse representation under D, 
or equivalently for any x g range(DT), where T is such that 
\T\ < k. 

As in the representation case, a connection between the D- 
RIP and the D-Spark can be established. 

Proposition 3.4 Given a matrix M and sparsity k, if 5^ < 1 
then k < D-Spark(M) . 

Proof: Requiring 6^ < 1 implies that for any vector x G 
range(DT) such that |r| < fc it holds that ||Mx||2 > (1 - 
(5P) ||x||2 > 0, hence Mx ^ . The last is equivalent to 
requiring Null(M) n rangc(DT) ~ {0} for any support 
set T such that |T| < fc, which is exactly equivalent to D- 
Spark(M) > fc. □ 



Having the definition of the D-RIP we present a stability 
guarantee for the signal recovery that appears in llT8][T9l . 

Theorem 3.5 Let y = Mxo + e where ||e||2 < e, xq has a 
k-sparse representation cxq under D, and M satisfies the D- 
RIP condition with (5^,. IfS^j^ < 1 then recovering the signal 
using Q, where the recovered signal is x = Da, is stable: 



xo - X 



< 



2e 



"2k 



(8) 



Proof Since ckq, the representation of Xq, is a feasible solu- 
tion to (O we have that ||a||Q < ||ao|lo ^ ^- Thus, xo — x is 
a signal that has a 2fc-sparse representation. According to the 
constraint in ^ we also have ||y — Mxjjg < e. Using the D- 
RIP, the triangle inequality, and the fact that j|y — Mxoj|2 
e as well, we get 



< 



Fig. 1. Representation recovery rate (left) and signal recovery 
rate (right). Color attribute: fraction of realizations in which 
£i minimization achieves a perfect recovery. 

4. PRACTICAL RECONSTRUCTION 

With the above observations, we turn to check the recovery 
performance of the ^i-minimization problem - a relaxed ver- 
sion that replaces the £q with £i-norm. In our experiment we 
consider the noiseless scenario, and since solving (|4) is NP- 
hard ll22l . the £i relaxation is inevitable when dealing with 
practical problems. We perform a simple synthetic test for 
signals that are sparse under a dictionary which is highly co- 
herent and with hnear dependencies between its columns. We 
generate a dictionary D = [Di, D2] where Di, D2 G M'*^'', 
d=1000, Di contains sparse columns with 2 non-zero en- 
tries which are 1 or —1 with probability 0.5 and D2 contains 
columns which are linear combinations of random 3 columns 
from Di with random zero-mean white Gaussian weights. 
Each entry of the measurement matrix M e ]]j'nxrf 
tributed according to a normal Gaussian distribution, where 
m=['-fd\ and 7 is the sampling rate - a value in the range 
(0, 1]. We set fc to be [pm\ {p ^ 1) and measure the recovery 
rate of the representation a and the signal x for various values 
of7e {0.1,0.2,...,l}andpe {0.02, 0.04, 0.2}. 

Figure [TIpresents the recovery performance over 100 real- 
izations per each parameter setting. Similar to the uniqueness 
results for the representation, the reconstruction fails almost 
always since D contains many linear dependencies (note that 
in our case Spark(MD) = 4). However, though we fail in 
reconstructing the representations, in most cases the recovery 
of the signal succeeds. Note that even in the case where m=d 
(7=1), we get a very low recovery rate for the representations 
and this is due to the non-uniqueness of any representation 
with cardinality beyond 1. The recovery rate on the bottom 
left part of the representation diagram is better than the other 
parts because for very small values of fc the chance to choose 
the wrong representation decreases significantly. 



Ilxo - ilia < 
1 

< 



1 



"2k 



1 - 



||M(xo-x)|| 



Mx, 



■0II2 



which is the stated result. 



|y-Mx||2) < 



2e 



(9) 5. CONCLUSION 



In this work we studied the £o-synthesis problem for signal re- 
covery in the case of a general dictionary D. We have shown 
that in the case where D contains linear dependencies signal 
recovery is still theoretically plausible. We derived theoreti- 
□ cal conditions for uniqueness of the solution in the noiseless 



"2k 



case and stability for the noisy case. In addition, empirical re- 
construction results were presented for the ^i-synthesis mini- 
mization problem, demonstrating that for this problem signal 
recovery is achievable in the case where D is highly coherent. 
These results motivate looking for theoretical guarantees for 
the signal recovery using the -relaxation in the same spirit 
of those developed in the representation case Il20l l5ll7l. a topic 
that would be covered by our upcoming research. 
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